Concurrent validity of the Osteopathic General Surgery In-Service Examination.

CONTEXT Performance on the Osteopathic General Surgery In-Service Examination (ISE) has been shown to improve over time for osteopathic general surgery residents. The training level-specific concurrent validity of the ISE, however, has not been evaluated. OBJECTIVE To investigate whether residents' scores will improve as they move from level 1 through level 5 of the ISE. METHODS In this retrospective study, performance on the ISE was obtained from the American College of Osteopathic Surgeons for all of the osteopathic general surgery residency programs from 2008 through 2012. The weighted raw score and standardized score performance mean and standard deviation were determined across training levels. One-way t tests were performed between residency years and ISE scores. Parametric statistics were calculated with α set to .05. RESULTS The authors evaluated 1952 examinations during the study period. Of the 49 programs screened, 33 (67.3%) met inclusion criteria for the present study. Analysis of variance tests showed that there was significant variation in raw and standardized outcomes between residency levels (both P<.001). One-tailed t tests for both raw and standardized outcomes showed that all scores' differences between examinee levels were statistically significant (P<.001), with the exception of raw scores between level 4 and level 5 examinees (P=.20). CONCLUSION There is near-uniform concurrent validity of the ISE by osteopathic general surgery training level. This psychometric characteristic supports the construct validity of this standardized test.